Binary Classification Model for Parkinson's Disease Using Python¶
David Lowe¶
April 24, 2019¶
Template Credit: Adapted from a template made available by Dr. Jason Brownlee of Machine Learning Mastery. [https://machinelearningmastery.com/]
SUMMARY: The purpose of this project is to construct prediction model using various machine learning algorithms and to document the end-to-end steps using a template. The Parkinson's Disease dataset is a binary classification situation where we are trying to predict one of the two possible outcomes.
INTRODUCTION: The data used in this study were gathered from 188 patients with PD (107 men and 81 women) with ages ranging from 33 to 87 at the Department of Neurology in Cerrahpasa Faculty of Medicine, Istanbul University. The control group consists of 64 healthy individuals (23 men and 41 women) with ages varying between 41 and 82. During the data collection process, the microphone is set to 44.1 KHz and following the physician's examination, the sustained phonation of the vowel /a/ was collected from each subject with three repetitions.
ANALYSIS: The baseline performance of the machine learning algorithms achieved an average accuracy of 81.58%. Two algorithms (Extra Trees and Stochastic Gradient Boosting) achieved the top accuracy metrics after the first round of modeling. After a series of tuning trials, Extra Trees turned in the top overall result and achieved an accuracy metric of 88.09%. By using the optimized parameters, the Extra Trees algorithm processed the testing dataset with an accuracy of 87.22%, which was just slightly below the prediction accuracy using the training data.
CONCLUSION: For this iteration, the Extra Trees algorithm achieved the best overall results using the training and testing datasets. For this dataset, Extra Trees should be considered for further modeling or production use.
Dataset Used: Parkinson's Disease Classification Data Set
Dataset ML Model: Binary classification with numerical attributes
Dataset Reference: https://archive.ics.uci.edu/ml/datasets/Parkinson%27s+Disease+Classification
The project aims to touch on the following areas:
- Document a predictive modeling problem end-to-end.
- Explore data cleaning and transformation options
- Explore non-ensemble and ensemble algorithms for baseline model performance
- Explore algorithm tuning techniques for improving model performance
Any predictive modeling machine learning project genrally can be broken down into about six major tasks:
- Prepare Problem
- Summarize Data
- Prepare Data
- Model and Evaluate Algorithms
- Improve Accuracy or Results
- Finalize Model and Present Results
import math
import numpy as np
import pandas as pd
import os
import sys
import shutil
import urllib.request
import zipfile
import warnings
import smtplib
from datetime import datetime
from email.message import EmailMessage
from matplotlib import pyplot
from pandas import read_csv
from pandas import get_dummies
from pandas.plotting import scatter_matrix
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.model_selection import KFold
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_auc_score
# from sklearn.metrics import mean_squared_error
from sklearn.pipeline import Pipeline
from sklearn.externals.joblib import dump
from sklearn.externals.joblib import load
from sklearn.feature_selection import RFE
from imblearn.over_sampling import SMOTE
from imblearn.combine import SMOTEENN
from imblearn.combine import SMOTETomek
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.ensemble import BaggingClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.ensemble import GradientBoostingClassifier
from imblearn.ensemble import BalancedRandomForestClassifier
from imblearn.ensemble import RUSBoostClassifier
from imblearn.ensemble import BalancedBaggingClassifier
from xgboost import XGBClassifier
# Create one random seed number for reproducible results
seedNum = 888
1.b) Set up the email notification function¶
def email_notify(msg_text):
sender = os.environ.get('MAIL_USERNAME')
password = os.environ.get('MAIL_PASSWORD')
receiver = os.environ.get('MAIL_RECEIVER')
if sender==None or password==None or receiver==None :
sys.exit("Incomplete email setup info. Script Processing Aborted!!!")
msg = EmailMessage()
msg.set_content(msg_text)
msg['Subject'] = 'Notification from Python Binary Classification Script'
msg['From'] = sender
msg['To'] = receiver
server = smtplib.SMTP('smtp.gmail.com', 587)
server.starttls()
server.login(sender, password)
server.send_message(msg)
server.quit()
email_notify("Library and Data Loading has begun! "+datetime.now().strftime('%a %B %d, %Y %I:%M:%S %p'))
1.c) Load dataset¶
startTimeScript = datetime.now()
dataset_path = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00470/pd_speech_features.rar'
dest_file = os.path.basename(dataset_path)
if (os.path.isfile(dest_file) == False) :
print('Downloading ' + dataset_path + ' as ' + dest_file)
with urllib.request.urlopen(dataset_path) as in_resp, open(dest_file, 'wb') as out_file:
shutil.copyfileobj(in_resp, out_file)
print(dest_file + 'downloaded!')
# print('Unpacking ' + dest_file)
# with zipfile.ZipFile(dest_file, 'r') as zip_ref:
# zip_ref.extractall('.')
# print(dest_file + 'unpacked!')
inputFile = "pd_speech_features.csv"
xy_original_df = read_csv(inputFile, sep=',', header=1)
xy_original_df.rename(columns={'class': 'targetVar'}, inplace=True)
xy_original_df.drop(columns=['id'], inplace=True)
xy_original_df.head()
# Use variable totCol to hold the number of columns in the dataframe
totCol = len(xy_original_df.columns)
# Set up variable totAttr for the total number of attribute columns
totAttr = totCol-1
# targetCol variable indicates the column location of the target/class variable
# If the first column, set targetCol to 1. If the last column, set targetCol to totCol
# If (targetCol <> 1) and (targetCol <> totCol), be aware when slicing up the dataframes for visualization
targetCol = totCol
# Standardize the class column to the name of targetVar if required
# xy_original_df = xy_original_df.rename(columns={'outcome': 'targetVar'})
# We create training datasets (xy_train_df, x_train_df, y_train_df) for various visualization and cleaning/transformation operations
# We create testing datasets (x_test_df, y_test_df) for various visualization and cleaning/transformation operations
# Use 70% of the data to train the models and the remaining for testing/validation
if targetCol == totCol:
x_original_df = xy_original_df.iloc[:,0:totAttr]
y_original_df = xy_original_df.iloc[:,totAttr]
else:
x_original_df = xy_original_df.iloc[:,1:totCol]
y_original_df = xy_original_df.iloc[:,0]
validation_size = 0.30
x_train_df, x_test_df, y_train_df, y_test_df = train_test_split(x_original_df, y_original_df, test_size=validation_size, random_state=seedNum)
xy_train_df = pd.concat([x_train_df, y_train_df], axis=1)
print("xy_original_df.shape: {} xy_train_df.shape: {}".format(xy_original_df.shape, xy_train_df.shape))
print("x_train_df.shape: {} y_train_df.shape: {}".format(x_train_df.shape, y_train_df.shape))
print("x_test_df.shape: {} y_test_df.shape: {}".format(x_test_df.shape, y_test_df.shape))
1.d) Set up the key parameters to be used in the script¶
# Set up the number of CPU cores available for multi-thread processing
cpu_num = None
# Set up the number of row and columns for visualization display. dispRow * dispCol should be >= totAttr
dispCol = 4
if totAttr % dispCol == 0 :
dispRow = totAttr // dispCol
else :
dispRow = (totAttr // dispCol) + 1
# Set figure width to display the data visualization plots
fig_size = pyplot.rcParams["figure.figsize"]
fig_size[0] = dispCol*4
fig_size[1] = dispRow*4
pyplot.rcParams["figure.figsize"] = fig_size
# Set the warning message filter
warnings.filterwarnings("ignore", category=FutureWarning)
email_notify("Library and Data Loading completed! "+datetime.now().strftime('%a %B %d, %Y %I:%M:%S %p'))
Section 2 - Summarize Data¶
To gain a better understanding of the data that we have on-hand, we will leverage a number of descriptive statistics and data visualization techniques. The plan is to use the results to consider new questions, review assumptions, and validate hypotheses that we can investigate later with specialized models.
email_notify("Data Summarization and Visualization has begun! "+datetime.now().strftime('%a %B %d, %Y %I:%M:%S %p'))
2.a) Descriptive statistics¶
2.a.i) Peek at the data itself.¶
xy_train_df.head()
2.a.ii) Dimensions of the dataset.¶
xy_train_df.shape
2.a.iii) Types of the attributes.¶
xy_train_df.info()
xy_train_df.dtypes
2.a.iv) Statistical summary of all attributes.¶
xy_train_df.describe()
2.a.v) Count missing values.¶
print('Number of NaN in the dataframe columns:\n', xy_train_df.isnull().sum())
print('Total number of NaN in the dataframe: ', xy_train_df.isnull().sum().sum())
2.a.vi) Summarize the levels of the class attribute.¶
xy_train_df.groupby('targetVar').size()
2.b) Data visualizations¶
2.b.i) Univariate plots to better understand each attribute¶
# Histograms for each attribute
x_train_df.hist(layout=(dispRow,dispCol))
pyplot.show()
# Density plot for each attribute
x_train_df.plot(kind='density', subplots=True, layout=(dispRow,dispCol))
pyplot.show()
# Box and Whisker plot for each attribute
x_train_df.plot(kind='box', subplots=True, layout=(dispRow,dispCol))
pyplot.show()
